Depiction system

ABSTRACT

A depiction system for generating a real time correlated depiction of movements of a surgical tool for uses in minimally invasive surgery is described. In an embodiment the system includes a computer system, 3D surface data generation means and position data generation means for obtaining real time spatial position data of at least a part of the surgical tool. The 3D surface data generation means or the position data generation is adapted for providing surface position data of at least the target area. The computer system is programmed for determining depiction data representing a depiction of the real time relative spatial position(s) of the surgical tool onto at least a portion of the surface contour of the surface section of the minimally invasive surgery cavity.

This is a continuation of International Application PCT/DK2016/050180,with an international filing date of Jun. 13, 2016 and claiming priorityfrom PA 2015 70642 DK of Oct. 9, 2015. International ApplicationPCT/DK2016/050180 is hereby incorporated by reference.

TECHNICAL FIELD

The invention relates to a depiction system for generating a real timedepiction of movements of a surgical tool during a minimally invasivesurgery and adapted for in real time providing a surgeon withinformation, such as information about the position of the surgicaltool.

BACKGROUND ART

Minimally invasive surgery has been used increasingly in recent yearsdue to the benefits compared to conventional open surgery as it reducesthe trauma to the patient tissue, leaves smaller scars, minimizespost-surgical pain and enables a faster recovery of the patient.

For example, in laparoscopic surgery which is a typical form ofminimally invasive surgery the surgeon accesses a body cavity, such asthe abdominal or pelvic cavity, through a series of small incisions. Alaparoscope is inserted through an incision, and conventionallyconnected to a monitor, thereby enabling the surgeon to see the insideof the abdominal or pelvic cavity. In order to perform the surgicalprocedure, surgical instruments are inserted through other incisions. Inaddition, the body cavity around the surgical site is inflated with afluid, preferably gas e.g. carbon dioxide in order to create an ‘air’space within the cavity to make space for the surgeon to view thesurgical site and move the laparoscopic instruments.

Invasive surgery procedures are generally performed through openings ina patient's skin—often relatively small openings—and the surgical siteis visualized for the surgeon by inserting a camera, such as anendoscope into the body cavity and displaying the images on a screen.

In order to improve the vision for surgeon, in particular to make iteasier for the surgeon to determine the sizes of various organs,tissues, and other structures in a surgical site, several in-situsurgical metrology methods have been provided in the prior art.Different types of optical systems have been applied to provide animproved vision of the surgical site, which is approaching a 3D vision.

US 2013/0296712 describes an apparatus for determining endoscopicdimensional measurements, including a light source for projecting lightpatterns on a surgical sight including shapes with actual dimensionalmeasurements and fiducials, and means for analyzing the projecting lightpatterns on the surgical site by comparing the actual dimensionalmeasurements of the projected light patterns to the surgical site.

WO 2013/163391 describes at system for generating an image, which thesurgeon can use for measuring the size of or distance between structuresin the surgical field by using an invisible light for marking a patternto the surgical field. The system comprises a first camera; a secondcamera; a light source producing light at a frequency invisible to humaneye; a dispersion unit projecting a predetermined pattern of light fromthe invisible light source; an instrument projecting the predeterminedpattern of invisible light onto a target area; a band pass filterdirecting visible light to the first camera and the predeterminedpattern of invisible light to the second camera; wherein the secondcamera images the target area and predetermined pattern of invisiblelight, and computes a three-dimensional image.

US2008071140 discloses an endoscopic surgical navigation systemcomprises a tracking subsystem to capture data representing positionsand orientations of a flexible endoscope during an endoscopic procedure,to allow co-registration of live endoscopic video with intra-operativeand/or pre-operative scan images. Positions and orientations of theendoscope are detected using one or more sensors and/or othersignal-producing elements disposed on the endoscope.

US2010268067 disclose methods, systems, devices, and computer-readablemedia for image guided surgery for allowing a physician to use multipleinstruments for a surgery and simultaneously provide image-guidance datafor those instruments.

US2011069159 discloses a system for orientation assistance and displayof an instrument that is inserted or present in the natural orartificially produced hollow cavity (human, animal, object), and that isequipped with one or more sensor units. Multiple measurements of the 3Dposition of the instrument equipped with one or more sensor units areperformed by positioning a measuring system, so that a preciseorientation and positioning of the instrument in the body can becomputed. The 3D position data are used to compute a virtual image ofthe instrument synchronously. The virtual images are then eitherprojected directly in exact position onto the body surface of a personor combined in a body surface image (real video camera image of thepatient) onto a monitor or superimposed (virtual or augmented reality).

It has also been suggested to generate augmented reality vision ofsurgery cavities for providing an improved view of internal structuresof patient organs of to determine minimally distance to a cavity surfaceor organ of a patient. Such systems are described in the articles“Augmented reality in laparoscopic surgical oncology” by StéphaneNicolau et al. Surgical Oncology 20 (2011) 189-201 and “An effectivevisualization technique for depth perception in augmented reality-basedsurgical navigation” by Choi Hyunseok et al. The international journalof medical robotics and computer assisted surgery, 2015 May 5. doi:10.1002/rcs.1657.

US2014163359 discloses a surgical tracking system for assisting anoperator to perform a laparoscopic surgery of a human body. Thedsurgical tracking system comprising: a) at least one endoscope adaptedto acquire real-time images of a surgical environment within said humanbody; b) a maneuvering subsystem adapted to control the spatial positionof said endoscope during said laparoscopic surgery; and, c) a trackingsubsystem in communication with said maneuvering subsystem, adapted tocontrol the maneuvering system so as to direct and modify the spatialposition of said endoscope to a region of interest. The system generatesreal life correlated images of movement of a surgical tool.

DISCLOSURE OF INVENTION

In an embodiment of the depiction system of the invention the systemprovides an alternative system for generating good visibility of atleast a part of a body cavity during minimally invasive surgery inparticular with respect to providing good visual information to thesurgeon about the position of a surgical instrument relative to thesurgical site.

The depiction system of the invention has shown to be capable ofproviding a surprisingly good visibility to a user, such as a surgeon ora person training for performing a surgery. In an embodiment the systemof the present invention aims to provide a depiction comprising aprojection of the position of a surgical tool rather than imaging thesurgical tool. Thereby the user can be provided with a large amount ofinformation in one single depiction. Whereas one single depiction canprovide the use with large amount of information, the invention alsocomprises a system generating two or more depictions or images asfurther described below.

The depiction system of the invention is suitable for generating a realtime correlated depiction of movements of a surgical tool.

The system comprising

-   -   a computer system configured for being in data communication        with a display unit,    -   3D surface data generation means for providing the computer        system with three-dimensional (3D) data representing at least        one surface section in 3D space of a minimally invasive surgery        cavity, wherein the surface section comprises a target area, and    -   position data generation means for obtaining real time spatial        position data of at least a part of the surgical tool and for        transmitting the obtained spatial position data to the computer        system.

At least one of the 3D surface data generation means and the positiondata generation means comprises surface position data generation meansfor providing surface position data of at least the target area.

The computer system is programmed for

-   -   determining a 3D surface contour of at least a part of the        target area of the surface section of the minimally invasive        surgery cavity using the 3D data,    -   determining real time spatially position(s) of the surgical tool        relative to at least a part of the target area of the at least        one surface section using the spatial position data and the        surface position data,    -   calculating depiction data representing a depiction of the real        time relative spatial position(s) of the surgical tool onto at        least a portion of the surface contour of the surface section of        the minimally invasive surgery cavity, and for

transmitting the depiction data to the display unit for real timecorrelated depiction of movements of the surgical tool.

The computer system may comprise one single computer or a plurality ofcomputers in data connection, wireless, by wire and/or via the internet.

The minimally invasive surgery cavity may be a patient or an artificialcavity. In general, an artificial cavity is useful in a trainingprocedure. In other procedures as well as in supervised procedures ofrelatively unexperienced surgeons the minimally invasive surgery cavityis advantageously a minimally invasive surgery cavity of a patient suchas a human or an animal.

The minimally invasive surgery cavity may in principle be any kind ofbody cavities including naturally occurring cavities as well as cavitiesformed by expansion of a non-luminal part of a body (pseudo cavity).Examples of suitable minimally invasive surgery cavities includes anabdominal cavity (the area between the diaphragm and the brim of thepelvis), a chest cavity, abdominopelvic cavity (contains both theabdominal and pelvic cavities), cranial cavity (contains the brain),diaphragm (muscle that separates the thoracic and abdominopelviccavities), mediastinum (central area within the thoracic cavity), pelviccavity (contains the reproductive organs and urinary bladder),pericardial cavity (contains the heart), pleural cavity (contains thelungs), thoracic cavity (enclosed by the ribs, sternum, and vertebralcolumn), vertebral cavity (contains the spinal cord), etc.

Usually the minimally invasive surgery procedure includes that thesurgeon provides access to the surgery site by an incision and apply acannula (sometimes also called a sleeve) to provide an access portthrough the incision. The cannula functions as a portal for thesubsequent placement of a surgical instrument comprising a surgicaltool. The term ‘surgical tool’ is herein used to designate the distalpart of a surgical instrument adapted to be inserted into the minimallyinvasive surgery cavity. A surgical instrument usually has a distal endand a proximal end and comprises a handle portion at its proximal end, asurgical tool at its distal end and a body portion connecting the handleportion to the surgical tool. The surgical tool may for example begraspers, scissors, staplers, etc. In an embodiment the body portion ofthe surgical instrument is considered as a part of the surgical tool.For examples sensors mounted to the surgical tool portion may be mountedto the body portion of the surgical instrument.

The cannula usually comprises one or more seals to seal against gasslip-out and to accommodate an instrument. After the cannula has beeninserted, the minimally invasive surgery cavity is usually enlarged byblowing a gas into the cavity thereby providing a cavity sufficientlylarge for performing the minimally invasive surgery procedure.

Often the surface of the minimally invasive surgery cavity is muchcurved. The term ‘target area’ of the surface of the minimally invasivesurgery cavity is herein used to designate the area which the surgeonhas focus on, and the target area may advantageously comprise a surgerysite and/or a surface area which potentially could be in risk of damageduring the surgery procedure for example a critical structure, such avene structure. The depiction data representing said depiction of saidreal tile spatial positions of the surgical tool onto at least a portionof the surface contour of the surface section of the minimally invasivesurgery, where the portion of the surface contour need not comprise thesurface contour of the target area constantly during a minimallyinvasive surgery procedure. It is sufficient that the contour of thetarget area forms part of the portion of the surface contour when thesurgical tool is approaching the target area.

In an embodiment depiction system is configured for determining realtime spatial position data comprising determining a distance from thesurgical tool to a critical structure of the surface section.

The phrase “real time” is herein used to mean the time it requires thecomputer to receive and process constantly changing data optionally incombination with other data, such as predetermined data, reference data,estimated data which may be non-real time data such as constant data ordata changing with a frequency of above 1 minute to thereby provide adepiction of corresponding actual changes as they occur or within up to5 seconds, preferably within 1 second, more preferably within 0.1 secondof occurrence.

The terms distal and proximal should be interpreted in relation to theorientation of the surgical tool i.e. the distal end of the surgicaltool is the part of the surgical tool farthest from the incision throughwhich the surgical instrument comprising the surgical tool is inserted.

The phrase “distal to” means “arranged at a position in distal directionto the surgical tool, where the direction is determined as a straightline between a proximal end of the surgical tool to the distal end ofthe surgical tool. The phrase “distally arranged” means arranged distalto the distal end of the surgical tool.

The term “substantially” should herein be taken to mean that ordinaryproduct variances and tolerances are comprised.

The term “about” is generally used to ensure that what is withinmeasurement uncertainties are include. The term “about” when used inranges, should herein be taken to mean that what is within measurementuncertainties are included in the range.

It should be emphasized that the term “comprises/comprising” when usedherein is to be interpreted as an open term, i.e. it should be taken tospecify the presence of specifically stated feature(s), such aselement(s), unit(s), integer(s), step(s) component(s) and combination(s)thereof, but does not preclude the presence or addition of one or moreother stated features.

Throughout the description or claims, the singular encompasses theplural unless otherwise specified or required by the context.

The portion, i.e. the area extension of the surface contour which iscomprised in the depiction data calculation may in principle have anysize larger than zero. In order to provide a depiction containingdesirably large amount of information for the user, the portion of thesurface contour advantageously has a size which is sufficiently large toreflect the surface contour of the portion. In an embodiment the portionof the surface contour has a size of at least about 5 mm², such as atleast about 10 mm², such as at least about 1 cm2, such as at least about5 cm², such as at least about 10 cm², such as at least about 25 cm²determined as the maximal size of a projection of the surface contouronto a plan—i.e. the plan is selected as the plan providing the largestprojected surface. In an embodiment the portion of the surface contourhas a size as least as large as the target area and/or as least as largeas the surgical site.

Advantageously the portion of the surface contour which is comprised inthe depiction data calculation has a size which is dynamically changingin dependence of the real time position data, Thereby the depiction mayreveal any movements of the surgical tool.

In an embodiment the portion of the surface contour is increasing withincreasing distance between the spatial position the part of thesurgical tool and the surface section.

In an embodiment the 3D surface data generation means and/or theposition data generation means is/are part of the depiction system andadvantageously the 3D surface data generation means and/or the positiondata generation means is/are controlled by the computer system. Thecomputer system is preferably programmed to control the operation of the3D surface data generation means and/or the position data generationmeans.

The 3D surface data generation means may be any means capable ofproviding the computer system with the 3D surface data, such as datafrom a database, data supplied to the computer system by a user, e.g. inform of pre-measured data e.g. by scanning e.g. CT scanning e.g. MR orreal-time measured data.

In an embodiment the 3D surface data generation means comprises a 3Ddatabase system. The 3D database system comprises at least one 3D dataset for at least one surface section of each of a plurality ofclassified minimally invasive surgery cavities, wherein each 3D data setis associated to the respective surface section(s) of at least one ofthe classified minimally invasive surgery cavities. Preferably thecomputer is configured for acquiring a 3D data set for at least onesurface section of a classified minimally invasive surgery cavity.

The classified minimally invasive surgery cavities are preferablyrepresented by their respective classification. The classification mayin principle comprise any classification data that describes theclassified minimally invasive surgery cavities e.g. type of cavity, ageof patient, weight of patient, gender of patient, height, bodycircumferential dimension(s) or any combinations comprising theabovementioned or any other specific or average or level of patientdata.

In an embodiment the computer is programmed to receive instructions forselecting a 3D data set and based on an instruction to acquire theselected 3D data set from the 3D database. The instruction mayadvantageously comprise classification data. The instruction may forexample be given to computer system via a digital user interface and/ororally by a user.

Unless otherwise stated, the term “user” is herein used to include anyuser, such as a surgeon, an assisting surgeon, a supervisor, a computerprogrammed to act as supervisor, an artificially intelligent computerand/or a robot, where a robot is programmed to perform at least onesurgeon duty, e.g. to perform or assist in a minimally invasive surgeryprocedure.

In an embodiment the surgeon is a robotic surgeon.

In an embodiment the visual information means optical data as receivedby a robotic surgeon and representing a surface section of the minimallyinvasive surgery cavity.

In an embodiment the computer is programmed to receive instructionscomprising classification data for selecting several 3D data sets andbased on this instruction to acquiring selected 3D data sets and processthe acquired 3D data sets to determine the resulting 3D data which isapplied to determine the 3D surface contour of the surface section ofthe minimally invasive surgery cavity.

In an embodiment the computer is programmed to receive instructionscomprising classification data for selecting several 3D data sets andbased on this instruction to acquiring selected 3D data sets andprocessing the acquired 3D data sets to determine the resulting 3D datawhich is applied to determine the 3D surface contour of the surfacesection of the minimally invasive surgery cavity.

The database system is data connectable or in data connection with thecomputer e.g. by wireless connection, by wire connection, via theinternet and/or by being loaded onto a memory of the computer system.

The database system may continuously be build up or updated by addingnew 3D data sets to the database.

In an embodiment the database system is loaded into a memory in datacommunication with or being a part of the computer system.

The 3D data sets of the database may be obtained by any methods and willusually be 3D data sets from previously performed minimally invasivesurgery procedures and/or estimated and/or calculated 3D data setsand/or 3D data sets obtained by a scanning such as described furtherbelow.

Advantageously one or more of the 3D data sets comprises estimated 3Ddata, calculated 3D data, measured 3D data or any combination thereof.Preferably the 3D data sets each are associated to a patientcharacteristic, such as a patient characteristic comprising age, gender,weight, height body circumferential dimension(s) or any combinationscomprising the abovementioned.

The estimated 3D data, calculated 3D data and/or measured 3D data of the3D data sets may be obtained or generate by any methods, such as methodsknown from prior art and/or methods described herein for obtaining 3Ddata a surface section of a minimally invasive surgery cavity. Inparticularly ultra sound based methods, CT scanning and/or MRI aresuitable as well as the prior art methods as disclosed in the articles“Augmented reality in laparoscopic surgical oncology” by StéphaneNicolau et al. Surgical Oncology 20 (2011) 189-201 and “An effectivevisualization technique for depth perception in augmented reality-basedsurgical navigation” by Choi Hyunseok et al. The international journalof medical robotics and computer assisted surgery, 2015 May 5. doi:10.1002/rcs.1657.

In an embodiment the 3D surface data generation means comprises a userinput data connection for feeding 3D data for a selected patient ontothe computer. The computer is configured for receiving such user input3D data, optionally by comprising a reader configured for reading imagese.g. 2D and/or 3D and comprising software generating the 3D data fromthe read images. Advantageously the user input 3D data is added as 3Ddata set to the database system.

In an embodiment the 3D surface data generation means comprises a 3Dsurface sensor system for determining at least a part of the 3D data forat least the surface section of the minimally invasive surgery cavityand transmitting means for transmitting the determined for 3D data tothe computer, optionally depiction system is configured for addingdetermined 3D data as 3D data set to the database system. Preferably thecomputer system is configured for adding determined 3D data receivedfrom the 3D surface sensor system as 3D data set to the database system.

In an embodiment the sensor system is adapted for determining at least apart of the 3D data for at least the surface section of the minimallyinvasive surgery cavity and the sensor system is further configured forgenerating at least one of pre-operative data and/or intra-operativedata.

The transmitting means may be any means suitable for transmitting said3D data, preferably in digital form. Suitable transmitting meansincludes USB key transmitting means, wire based transmitting meansand/or wireless transmitting means, such as Bluetooth.

The surface sensor system may comprise any type of surface sensorsuitable for determining 3D data of the surface section. In a preferredembodiment the 3D surface sensor system comprises at least one localreference sensor, preferably for spatial reference. Advantageously theat least one local reference sensor is adapted to be positioned on apatient and/or on a support (surgery table) for a patient.

In an embodiment there are a plurality of reference sensors, thereference sensors are preferably configured to communicate to locate aposition of each other to thereby define an X.Y.Z dimensional space—e.g.as describes in US 2007/060098 or U.S. Pat. No. 6,631,271. In anembodiment the 3D surface sensor system is as described in “Spatial DataEstimation in Three Dimensional Distributed Wireless Sensor Networks”,by Karjee et al. Embedded Systems (ICES), 2014 International Conference3-4 Jul. 2014, IEEE ISBN 978-1-4799-5025-6.

In an embodiment the 3D surface data generation means comprises thesurface position data generation means for providing surface positiondata of the target area. The surface position data generation meansadvantageously comprises the at least one local reference sensordescribed above.

The 3D surface sensor system may advantageously comprise a 3D opticalsensor, an acoustic sensor, a magnetic sensor, an electric sensor, anelectromagnetic sensor or any combinations thereof.

In an embodiment the 3D surface sensor system comprises a surfacetracking system as described in “Augmented reality in laparoscopicsurgical oncology” by Stéphane Nicolau et al. Surgical Oncology 20(2011) 189-201.

The one or more sensors of the sensor system can in principle bepositioned anywhere as long as it is capable of sensing at least a partof the surface section. Advantageously the one or more sensors of thesensor system is/are part of the depiction system. Where the one or moresensors of the sensor system is/are fixed to or integrated with one ormore elements, this/these one or more elements advantageously form partof the depiction system.

In an embodiment at least one sensor is positioned on or is adapted tobe positioned on a trocar and/or cannula, the trocar and/orcannula ispreferably a part of the depiction system. In an embodiment at least onesensor is positioned on or is adapted to be positioned on a surgicalinstrument, preferably on a surgical tool of the surgical instrument foruse in minimally invasive surgery, the surgical instrument or thesurgical tool is preferably a part of the depiction system. In anembodiment at least one sensor is positioned on or is adapted to bepositioned on an endoscope for use in minimally invasive surgery, theendoscope is preferably a part of the depiction system. In an embodimentat least one sensor is positioned on or is adapted to be positioned on apatient e.g. in the minimally invasive surgery cavity, such as on asurface thereof or external on the surface of the skin of a patient. Inan embodiment at least one sensor is positioned on or is adapted to bepositioned on a sensor instrument which can be at least partly insertedinto the minimally invasive surgery cavity, the sensor instrument ispreferably a part of the depiction system. In an embodiment at least onesensor is adapted to be positioned external to the minimally invasivesurgery cavity e.g., to be manually handled or handled by a robot,preferably the external sensor (e.g. a scanning instrument—CT, NMR, MR,UV and/or IR scanner) and optionally the robot form part of thedepiction system.

In an embodiment the 3D surface sensor system comprises a 3D opticalsensor system comprising at least one optical source and at least oneoptical reader such as an image recorder comprising a 2D array ofdetectors, such as a camera, the 3D optical sensor system is preferablya binocular sensor system or a multiocular sensor system.

The optical source of the 3D optical sensor source may in principlecomprise any kind of optical source, such as an illumination source, aninvisible source—e.g. an IR source.

The optical source may be a coherent light source or an incoherent lightsource. Examples of optical sources includes a semiconductor lightsource, such as a laser diode and/or a VCSEL light source as well as anykind of laser sources including narrow bandwidth sources and broad bandsources.

In an embodiment the optical source has a band width (full width at halfmaximum—FWHM) of up to about 50 nm, such as from 1 nm to about 40 nm.Preferably the narrow band width of the optical source is about 25 nm orless, such as about 10 nm or less.

In an embodiment the optical source is a broad band light source such asa supercontinuum light source e.g. spanning at least an octave withinthe bandwidth range from 400 nm to 2600 nm. Above 2600 nm lighttransmitted in a silica fiber will be strongly attenuated.

In an embodiment the optical source is configured for emitting at leastone electromagnetic wavelength within the UV range of from about 10 nmto about 400 nm, such as from about 200 to about 400 nm.

In an embodiment the optical source is configured for emitting at leastone electromagnetic wavelength within the visible range of from about400 nm to about 700 nm, such as from about 500 to about 600 nm.

In an embodiment the optical source is configured for emitting at leastone electromagnetic wavelength within the IR range of from about 700 nmto about 1 mm, such as from about 800 to about 2500 nm.

In an embodiment the optical source comprises at least one wavelengthwithin the invisible range, such as the UV or the IR range.

Advantageously the optical source is tunable in wavelength and/or power.

The optical source is advantageously connected to or integrated in anendoscope.

In an embodiment the 3D optical sensor system comprise two opticalsources, preferably laser sources of equal or different wavelength(s)and being emitted towards the surface in different light beam angles(determined as the angle of the centremost ray of the respective lightbeams).

In an embodiment the 3D optical sensor system is based on USING BEAMPHASE interference determination e.g. as described in US2004/0145746.

In an embodiment the 3D optical sensor system may for example comprise aphase optic element (e.g. a diffractive optic element (DOE)), a spatiallight modulator, a multi-order diffractive lens, a holographic lens, aFresnel lens and/or a computer regulated optical element. In anembodiment the DOE is as described in US 2013/0038836 e.g. as shown inFIG. 1 and/or as described in section [0015] of US 2013/0038836.

In an embodiment the 3D optical sensor system comprises two cameras e.g.as described in WO 2013/163391.

In an embodiment the at least one optical source is configured foremitting an optical tracking light pattern which pattern when impingedonto and reflected and/or scattered from the surface reveal the contourof the surface to thereby provide 3D data to be recorded by the opticalreader. In this embodiment the 3D optical sensor system mayadvantageously be or comprise the apparatus described in US 2013/0296712and/or EP 2 586 34.

In an embodiment at least one of the optical pattern emitting source andthe recorder is positioned on an endoscope which asvantageously formspart of the system.

In an embodiment the optical source is adapted for emitting asubstantially stationary optical pattern onto at least a part of thesurface section which pattern when impinged onto and reflected and/orscattered from the surface reveal the contour of the surface to therebyprovide 3D data to be recorded by the optical reader and the opticalpattern emitting source optionally being positioned on an endoscopewhich is adapted to be held substantially stationary during theminimally invasive surgery procedure, such as on an endoscope alsocomprising the recorder.

In the following the term “reflected and/or scattered from a surface”should be interpreted to include any interactions with the relevantsurface which is readable by an optical recorder. The light in questionmay be visibly or invisible to the human eye e.g. including wavelengthsfrom below 300 nm to above 2 μm.

Advantageously the depiction system preferably comprises both astationary and a dynamic optical pattern source. The dynamic opticalpattern source is preferably in form of a light source adapted foremitting said dynamic optical pattern onto at least a part of thesurface section. The dynamic optical pattern is preferably dynamic bybeing mounted onto a surgical tool such that the reflected and/orscattered pattern (the pattern reflected and/or scattered from thesurface section), which may be recorded by the recorder is changing inrelation to movements of the surgical tool. Preferably the changes ofthe reflected and/or scattered pattern are correlated to the movementsof the surgical tool. The correlated movement of the pattern may e.g. beprovided as described in WO 15/124159 e.g. comprising the set comprisinga surgical instrument wherein the surgical tool as described hereinincludes the body and the surgical tool of the surgical instrument of WO15/124159. The correlated movement of the pattern may e.g. be providedas described in DK PA 2015 70483. The optical recorder may be mounted onthe surgical tool and/or on an endoscope or at any other suitable means.

The optical source may be temporarily or permanently fixed to thesurgical tool and the optical source preferably also form part of theposition data generation means.

By simultaneously use of the a stationary and a dynamic optical patternsource both the 3D data—or at least some of the 3D data optionallysupplemented with one or more 3D data set(s) from the 3D databasesystem—and the real time position data can be generated simultaneously.

In an embodiment the depiction system comprises a dynamic opticalpattern preferably dynamic by being mounted onto a surgical tool asdescribed above and simultaneously the system comprises an opticalrecorder mounted to the surgical tool or to an endoscope for collectingboth 3D data and position data and preferably orientation data.

In an embodiment the 3D surface sensor system comprises an acousticsensor system comprising at least one sonic wave emitter and at leastone sonic wave receiver. Preferably the acoustic sensor system comprisestwo or more sonic wave receivers, such as a 2D array of sonic wavereceivers e.g. piezo sonic wave receivers.

Advantageously the acoustic sensor system comprises an ultrasoundsensor. The 3D optical sensor system may e.g. be as described in any oneof U.S. Pat. No. 4,694,434 of US 2014/0204702.

In an embodiment the 3D surface sensor system comprises a magneticsensor system, such as a magnetic induction tomography sensor systemand/or a magnetic resonance tomography sensor and/or an electromagneticsensor system.

In an embodiment the 3D surface sensor system comprises an electricsensor system, such as a micro mechanical sensor system.

In an embodiment the 3D surface sensor system comprises an X-raycomputed tomography sensor (CT scanning sensor).

The position data generation means may be any kind of position datageneration means suitable for determining the real time spatial positionof at least a part of the surgical tool. Preferably the part of thesurgical tool includes at least the distal end of the tool.

In an embodiment the position data generation means for obtaining realtime spatial position data of at least a part of the surgical toolcomprises a position sensor system. The position sensor systempreferably comprises a 3D optical sensor, an acoustic sensor, a magneticsensor, an electric sensor, an electromagnetic sensor, an accelerometer,a gyroscope, a gravimeter an inertial navigation system a localpositioning system or any combinations thereof.

Advantageously the position sensor system is part of the depictionsystem.

In an embodiment the position sensor system comprises an inertialnavigation system. The inertial navigation system advantageouslyincludes at least a computer (e.g. of the computer system) and aplatform or module containing accelerometers, gyroscopes and/or othermotion-sensing devices. The inertial navigation system may initially beprovided with its position and velocity from another source (such as ahuman operator, a GPS satellite receiver, etc.), and thereafter computesits own updated position and velocity by integrating informationreceived from the motion sensors. The advantage of an inertialnavigation system is that it requires no external references in order todetermine its position, orientation, or velocity once it has beeninitialized.

In an embodiment the position sensor system comprising a sensor elementadapted to be or being physically connected to or integrated with thesurgical tool.

In an embodiment the position sensor system comprises a magnetic motioncapture systems comprising at least one sensor adapted to be or beingphysically connect to or integrated with the surgical tool to measurelow-frequency magnetic fields generated by a transmitter source.

In an embodiment the position sensor system comprises a MEMS sensormagnetic motion capture systems adapted to be or being physicallyconnect to or integrated with the surgical tool to measure return signalupon activation by a transponder.

In an embodiment the position sensor system comprises an acoustic sensorincluding at least one sensor mounted on integrated with the surgicaltool the surgical tool for increase accuracy—e.g. for directiondetermination.

The one or more sensors of the position sensor system may advantageouslybe positioned as described above for the sensor(s) of the 3D sensorsystem optionally with an additional sensor mounted on or integratedwith the surgical tool. Advantageously one or more sensors may be partof both the 3D sensor system and the position sensor system.

In an embodiment the position sensor system comprises at least one localreference sensor, preferably for spatial reference. The at least onelocal reference sensor is preferably adapted to be positioned on apatient, such as in the minimally invasive surgery cavity or on theouter skin of the patient and/or on a support (e.g. a surgery table) fora patient.

In an embodiment the position sensor system comprises a plurality ofreference sensors—preferably also being a part of the 3D surface sensorsystem. The reference sensors are preferably configured to communicateto locate a position of each other to thereby define an X.Y.Zdimensional space—e.g. as describes in US 2007/060098 or U.S. Pat. No.6,631,271. In an embodiment the position sensor system is based on thetechnology described in “Spatial Data Estimation in Three DimensionalDistributed Wireless Sensor Networks”, by Karjee et al. Embedded Systems(ICES), 2014 International Conference 3-4 July 2014, IEEE ISBN978-1-4799-5025-6.

In an embodiment the at least one reference sensor provides the localpositioning system, preferably the position sensor system comprises aplurality local reference sensors.

The position sensor system is advantageously configured for obtainingreal time spatial position data of the surgical tool and fortransmitting the obtained spatial position data to the computer system,wherein the real time spatial position data is on form of real timespatial position data in an X-Y-Z dimensional space.

In an embodiment the position sensor system comprises at least onedistance sensor, such as a short range laser based sensor e.g. asprovided by SICK AG, Germany.

A short range laser based distance sensor is operating by projecting alight beam spot onto a measurement object, e.g. using a laser diode. Bymeans of an optical receiver, the reflection is mapped onto a lightsensitive element (such as CMOS). Based on the position of the mappedlight spot and the 3D surface data, the distance to the surface sectioncan be determined.

Advantageously the real time spatial position data comprises positiondata correlated to the 3D data representing at least one surface sectionin 3D space, preferably the real time spatial position data comprisesdetermining a distance from the surgical tool or the at least one partof the surgical tool to the surface section, preferably to the targetarea of the surface section.

In an embodiment the real time spatial position data comprisesdetermining a distance e.g. using a short range laser based distancesensor from the surgical tool or at least one part of the surgicaltool—preferably its distal end—and to a point of the surface sectioncorrelated with an orientation direction of the tool, where theorientation direction of the tool is determined from an access openinginto the minimally invasive surgery cavity through which the tool isinserted and the farthest extension of the tool into the cavity. Thisorientation direction is also referred to as the longitudinal vectordirection of the surgical tool and/or where the surgical tool is notbended or bendable—the longitudinal direction of the surgical tool.Where the surgical tool is bended the longitudinal direction isdetermined as the direction of the distal part of the surgical tool fromthe bend to the most distal part of the tool.

In an embodiment the real time spatial position data comprisesdetermining a distance from the surgical tool or at least one part ofthe surgical tool—preferably its distal end—and to a point of thesurface section in distal direction to the surgical tool.

Advantageously the 3D surface sensor system and the position sensorsystem are at least partly integrated to form a combined 3D surface andposition sensor system. Preferably this 3D surface and position sensorsystem is a part of the depiction system.

In an embodiment the computer system is in data connection with the 3Dsurface data generation means and is configured for acquire the 3D datarepresenting at least one surface section in 3D space of a minimallyinvasive surgery cavity from the 3D surface data generation means.Preferably the computer system is configured for receiving instructionvia a user interface to acquire the 3D data and upon receiving suchinstructions to acquire the 3D data.

In an embodiment the computer system—for each user procedure—isconfigured to acquire the 3D data representing at least one surfacesection in 3D space of a minimally invasive surgery cavity in one singletime data package. In this embodiment the acquired 3D data representingat least one surface section in 3D space of a minimally invasive surgerycavity may be used in the whole minimally invasive surgery procedure,and the 3D surface data generation means is advantageously controlled bythe computer system to acquire the 3D data representing at least onesurface section in 3D space of a minimally invasive surgery cavity onlyonce in one data package.

In another embodiment the computer system—for each user procedure—isconfigured for acquiring the 3D data by consecutively acquiring 3D datapackages representing at least one surface section in 3D space of aminimally invasive surgery. The consecutively acquired 3D data packagespreferably comprises timely updated 3D data and/or 3D data representingat least one additional surface section in 3D space for each of theconsecutive acquired 3D data packages. Advantageously the operation ofthe 3D surface data generation means is controlled by the computersystem.

In an embodiment the computer system is in data communication with theposition data generation means and is configured for in real timeacquire the obtained real time spatial position data of at least a partof the surgical tool. Preferably the computer system is configured forreceiving instruction via a user interface to acquire the spatialposition data and upon receiving such instructions to acquire thespatial position data.

In an embodiment the computer system is configured for controlling the3D surface data generation means and the position data generation means.Preferably the computer is configured for receiving instruction via auser interface to conduct a user process and based on the instruction toacquire the required data from the 3D surface data generation means andthe position data generation means and for conducting the user procedureuntil a termination signal is transmitted to the computer.

Advantageously the system further comprises orientation data generationmeans for obtaining real time spatial orientation data of at least apart of the surgical tool and for transmitting the obtained spatialorientation data to the computer system. The computer system isadvantageously programmed for

-   -   determining real time spatially orientation(s) of the surgical        tool using the spatial orientation data,    -   calculating depiction data representing a depiction of the real        time spatial orientation(s) of the surgical tool onto at least a        portion of the surface contour of the surface section of the        minimally invasive surgery cavity, and for    -   transmitting the depiction data to the display unit for real        time correlated depiction of movements of the surgical tool.

Advantageously the computer system is configured to calculatingdepiction data representing the depiction of the real time relativespatial position(s) and the real time spatial orientation(s) of thesurgical tool onto at least a portion of the surface contour of thesurface section of the minimally invasive surgery cavity comprisesdepiction data representing an associated depiction of the real timerelative spatial position(s) and the real time spatial orientation(s) ofthe surgical tool onto at least a portion of the surface contour.

Thereby an effective depiction with a concentrated amount of informationcan be visualised for the user.

In an embodiment the associated depiction of the real time relativespatial position(s) and the real time spatial orientation(s) of thesurgical tool onto at least a portion of the surface contour comprises adepiction of the real time relative spatial position(s) onto the surfacecontour in a direction coordinated with the real time spatialorientation(s) of the surgical tool. The resulting depiction willthereby be a dynamically changing depiction and will thereby show evensmaller movements of the surgical tool including tilting movements whichwill bring very useful information to the user.

Advantageously the orientation data generation means comprises anorientation sensor connected to or integrated with the surgical tool. Inan embodiment the surgical tool is part of the depiction system andcomprises integrated at least one sensor.

In an embodiment the orientation data generation means comprises anorientation sensor connected to or integrated with the surgical tool.The orientation sensor preferably comprises at least one distancesensor, such as a short range laser based sensor described above. The atleast one direction sensor preferably being mounted to emit laser lightin distal direction relative to the surgical tool, preferably orientedsubstantially parallel to the longitudinal direction of the surgicaltool.

In an embodiment the position data generation means and the orientationdata generation means are at least partly integrated to a combinedposition data and an orientation data generation means. In practice itwill be simpler to have a combined position data and an orientation datageneration means.

In this embodiment the position data/position data generation means alsocomprises orientation data/orientation data generation means.

Preferably the depiction data representing a depiction of the real timerelative spatial position(s) of the surgical tool onto the determined 3Dsurface contour of the surface section of the minimally invasive surgerycavity comprises depiction data encoding a depiction of a dynamicpattern representation, a dynamic scaling of colours representation, adynamic schematic representation, a dynamic graphical representationand/or a dynamic augmented reality representation.

Advantageously the encoded depiction comprises a non-image-accuratedepiction of the surgical tool.

Whereas prior art systems heretofore have been focused on generating 3Dvision of a minimally invasive surgery area or a surgical instrumentwhich is an imitation of a real life vision of the minimally invasivesurgery area or a surgical instrument the system of the presentinvention provides a depiction comprising a correlated projection of thereal time position and movements and preferably real time orientation ofthe surgical tool rather than imaging the surgical tool as such. Byproviding the depiction of the real time relative spatial position(s) ofthe surgical tool onto the determined 3D surface contour of the surfacesection of the minimally invasive surgery cavity in form of depictiondata encoding a depiction of a dynamic pattern representation, a dynamicscaling of colours representation, a dynamic schematic representation, adynamic graphical representation and/or a dynamic augmented realityrepresentation, wherein the encoded depiction does not comprise anon-image-accurate depiction of the surgical tool, the resultingdepiction can comprise a very high concentration of useful informationto the user. For easy decoding of the depiction the user may be trainede.g. as described further later on. Advantageously the depiction alsoincludes sound as exemplified below.

In an embodiment the dynamic scaling of colours comprises a dynamicscaling of shades of one or more colours, such as hue colours.

The term shades are herein used to designate a gradation of a colourwith more or less black and/or brightness.

In an embodiment the depiction data comprises data encoding a depictionof a dynamic pattern representation, wherein the dynamic patternrepresentation comprises a depiction of a virtual pattern resembling anemitted light pattern impinged onto the determined 3D surface contour,wherein the virtual pattern preferably comprises arch shaped and/or ringshaped lines and/or a plurality of angled lines.

Advantageously the dynamic pattern representation comprises a depictionof the virtual pattern onto the determined 3D surface contour, such thatthe depiction comprises a dynamic modification of the virtual patternwherein the dynamic modification is correlated to the determined realtime spatially position(s) and thereby movements of the surgical tool.

The virtual pattern preferably resembling a light pattern comprising atleast spatially dispersed light beam fractions, such as an angular lightpath surrounding a light dot or a grid of lines, e.g. a crosshatchedpattern optionally comprising substantially parallel lines when emittedto a planar surface, and/or one or more angular, such as rectangularshapes e.g. square shaped, e.g. in an overlapping configuration, in aside by side configuration or concentrically arranged.

In an embodiment the dynamic pattern representation comprises adepiction corresponding to the light pattern reflected and/or scatteredfrom the surface of the surgery site when using the systems and methodsdisclosed in WO15124159 and/or in co-pending patent application DK PA2015 70483.

In an embodiment the depiction comprises a depiction of a dynamicscaling of colours representation, wherein the dynamic scaling ofcolours representation comprises a visual coloured representation of thedetermined real time spatially position(s) of the surgical tool relativeto the determined 3D surface contour wherein the colouring isdynamically modified in correlation to changes of the spatial positionand optional orientation caused by the movements of the surgical toolrelative to the determined 3D surface contour for example such that theshorter distance between the surgical instrument and the target area themore intensive red and/or less intensive green.

The depiction of a dynamic scaling of colours representation has shownto be very simple to decoding for most users and can comprise a largeamount of useful information. Advantageously the depiction systemscomprise means (a button or wireless setting means) for setting thecolour scaling and brightness. For some users the colour scaling is notsufficient e.g. due to colour blindness and in such situation otherdepiction are preferred e.g. in combination with the colour scalingrepresentation.

In an embodiment the visual coloured representation comprises a 2Dgraduated shading.

In an embodiment the depiction comprises a dynamic schematicrepresentation, wherein the dynamic schematic representation comprises adiagrammatic representation of the determined real time spatiallyposition(s) of the surgical tool relative to the determined 3D surfacecontour wherein the diagrammatic representation is dynamically modifiedin correlation to changes of the spatial position and optionalorientation caused by the movements of the surgical tool relative to thedetermined 3D surface contour.

In an embodiment the depiction comprises a depiction of a dynamicgraphical representation, wherein the dynamic graphical representationcomprises a graphical representation of the determined real timespatially position(s) of the surgical tool relative to the determined 3Dsurface contour wherein the graphical representation is dynamicallymodified in correlation to changes of the spatial position and optionalorientation caused by the movements of the surgical tool relative to thedetermined 3D surface contour.

In a preferred embodiment the depiction comprises a depiction of adynamic augmented reality representation, wherein the dynamic augmentedreality representation comprises an augmented reality representation ofthe determined real time spatially position(s) of the surgical toolrelative to the determined 3D surface contour wherein the augmentedreality representation is dynamically modified in correlation to changesof the spatial position and optional orientation caused by the movementsof the surgical tool relative to the determined 3D surface contour.

In an embodiment the augmented reality is a spatial augmented reality(SAR) wherein the display unit comprises one or more digital projectorsconfigured to display the depiction upon receipt of the calculateddepiction data e.g. onto a physical object(s), such as onto the patient,such as onto an artificial patient or a head-up display.

Advantageously the depiction comprises a sound depiction, such as abeeping sound where the tone and/or the beep-rate is correlated to therelative distance between the surgical tool and the surface section. Thebeeping sound may for example intensify as the surgical tool isapproaching the surface section e.g. to a point of the surface sectiondistal to the surgical tool in its longitudinal direction.

In an embodiment the depiction comprises a vibration depiction, such asa vibration of the surgical tool, such as of a proximal part of thesurgical tool adapted to be sensed by a surgeon holding or being mountedwith the surgical tool. The vibration may for example be activated incase the surgical tool is dangerously close to a critical structure ofthe surface section of the minimally invasive surgery cavity.

In an embodiment the calculated depiction data is configured for beingdisplaced on the display unit, wherein the display unit is selected froma 2D screen, a 3D screen, a projector system (such as an augmentedprojector system), head-up display, a wearable display unit, such as ahead mounted display (e.g. goggles) or any combinations thereof.

In an embodiment the calculated depiction data comprises pixel data fora pixel based display unit, such as a screen, the pixel data preferablycomprises red-blue-green pixel data, preferable for displaying highcolour or true colour comprising at least 16 bits per pixel (bpp), suchas at least 20 bpp.

The pixel data of the depiction data representing a depiction of thereal time spatial position(s) of the surgical tool onto the determined3D surface contour of the surface section of the minimally invasivesurgery cavity is preferably dynamically modified in correlation tochanges of the spatial position and optional orientation caused by themovements of the surgical tool relative to the determined 3D surfacecontour.

To obtain a high quality depiction the calculated depiction dataadvantageously comprises pixel data for a 2D display, such as a 2Ddisplay of at least about 1000 pixels, such as at least 10.000 pixelsoptionally comprising sub-pixels.

In an embodiment the display unit is part of the depiction system, thedisplay unit is preferably selected from a 2D screen, a 3D screen, aprojector system (such as an augmented projector system), head-updisplay, a wearable display unit, such as a head mounted display (e.g.goggles) or any combinations thereof.

To increase the visual perception of the user it is in an embodimentdesired that the depiction system comprises a real imaging systemconfigured for generating real imaging data for a real imaging of the atleast one surface section of the minimally invasive surgery cavity. Thereal imaging system is a 2D real imaging system, a 3D real imagingsystem, a virtual reality real imaging system, an augmented reality realimaging system or any combination thereof.

The term “real imaging” is herein used to mean an imaging where theimages show or images the surface section as it is in real life.

The real imaging system may for example be an endoscopic system as it iswell known today, e.g. an endoscope for inserting into the minimallyinvasive surgery cavity and comprising an illuminator for illuminationthe surface target area and a camera for requiring real images of thetarget area.

In an embodiment the real imaging system is an argumented reality realimaging system e.g. as describe in“Augmented reality in laparoscopicsurgical oncology” by Stéphane Nicolau et al. Surgical Oncology 20(2011) 189-201 and “An effective visualization technique for depthperception in augmented reality-based surgical navigation” by ChoiHyunseok et al. The international journal of medical robotics andcomputer assisted surgery, 2015 May 5. doi: 10.1002/rcs.1657.

Advantageously the real imaging system comprises an image recorder(camera) for acquiring at least one image of the at least one surface ofsection of the minimally invasive surgery cavity. The image recorderpreferably being a part of an endoscope, the image recorder preferablybeing a videoscope preferably acquiring real time images of the at leastone surface section of the minimally invasive surgery cavity.

The image recorder is for example a 2D recorder comprising a 2D array ofdetectors in form of pixel detectors, preferably the image recordercomprises at least 1000 pixels, such as at least 10.000 pixels,preferably the image recorder is a mega pixel recorder comprising atleast 1 mega pixels.

In an embodiment the image recorder is a 3D recorder comprising a 3Darray of detectors in form of pixel detectors, preferably the imagerecorder comprises at least 1000 pixels, such as at least 10.000 pixels,preferably the image recorder is a mega pixel recorder comprising atleast 1 mega pixels.

In an embodiment the image recorder or preferably the whole real imagingsystem constitutes also a part of the 3D surface data generation meansfor providing the computer system with the three-dimensional (3D) datarepresenting the at least one surface section in 3D space of theminimally invasive surgery cavity.

Preferably the depiction system is configured for transmitting the realimaging data to the display unit for providing a real imaging of the atleast one surface section of the minimally invasive surgery cavity, thereal imaging preferably comprises displaying the acquired image(s).

By displaying both the real imaging of the at least one surface sectionof the minimally invasive surgery cavity and the depiction of the realtime relative spatial position(s) of the surgical tool onto the surfacecontour of the surface section of the minimally invasive surgery cavitythe visual perception of the user is highly increased. Advantageouslydepiction system is configured for displaying the real imaging of the atleast one surface section of the minimally invasive surgery cavity andthe real time correlated depiction of movements of the surgical tool bya common display unit.

Advantageously the depiction system is configured for displaying thereal imaging of the at least one surface section of the minimallyinvasive surgery cavity and onto the real imaging of the at least onesurface section of the minimally invasive surgery cavity to display thedepiction of the real time relative spatial position(s) of the surgicaltool onto the surface contour of the surface section of the minimallyinvasive surgery cavity.

Preferably the depiction of the real time relative spatial position(s)of the surgical tool onto the surface contour of the surface section ofthe minimally invasive surgery cavity is projected onto the real imagingof the at least one surface section of the minimally invasive surgerycavity, the depiction of the real time relative spatial position(s) ofthe surgical tool onto the surface contour of the surface section of theminimally invasive surgery cavity is preferably at least partlytransparent for the real imaging of the at least one surface section ofthe minimally invasive surgery cavity. Thereby the user can on the samedisplay se both the real imaging of the at least one surface section ofthe minimally invasive surgery cavity and onto the real imaging of theat least one surface section of the minimally invasive surgery cavity todisplay the depiction of the real time relative spatial position(s) ofthe surgical tool onto the surface contour of the surface section of theminimally invasive surgery cavity in a correlated way, which furtherincrease the visual perception.

In an embodiment the depiction system comprises a robot controller forcontrolling a robot which preferably form part of the depiction system.The robot and the robot controller may in an embodiment be as describedin US 2009/0248041. The robot controller is in data connection with oris integrated with the computer system. In the following the robotcontroller is describe as being in data communication with the computersystem but it should be interpreted such that the computer controller aswell can be an integrated part of the computer system.

The robot controller being configured for receiving at least a part ofthe obtained or generated data preferably comprising said 3D surfacedata and said real time spatial position data and preferably said realtime orientation data. Optionally the robot controller is alsoconfigured for receiving derived data—i.e. data derived from the 3Dsurface data and said real time spatial position data and preferablysaid real time orientation data—such as the depiction data. The robotcontroller is preferably configured for controlling the robot forhandling the surgical tool for performing a minimally invasive surgeryprocedure.

In an embodiment the robot controller is configured for acquiring therequired data from the computer system respective data from saidcomputer system.

Advantageously the robot controller is configured for receivinginstructions from a supervisor, such as an assisting surgeon and/or arobot operator. The controller is preferably configured for receivinginstructions from the supervisor by data input via a digital userinterface and/or by oral instruction.

The robot controller is preferably configured to modify movements ofsaid surgical tool in response to the supervisor instructions. Thesupervisor instruction may for example instruct the robot to move thesurgical tool a cm to the right or a few mm distally in longitudinaldirection of the surgical tool or etc.

In an embodiment the depiction system is configured for simultaneously,via the robot controller, controlling the robot for handling thesurgical tool for performing a minimally invasive surgery procedure andfor transmitting said depiction data to the display unit. The depictionwill in this embodiment comprise a real time correlated depiction ofsaid movements of said surgical tool by said robot. The supervisor maykeep the robot under observation during its performing of the minimallyinvasive surgery via the depiction and thereby control that the robot isoperation sufficiently accurate or as explained, the supervisor maycorrect the robot by feeding instructions to the robot controller.

The computer system preferably comprises a memory and is advantageouslyconfigured for storing performance data sets, where each performancedata set comprises performance data associated with a minimally invasiveprocedure, such as for minimally invasive procedure that has beenperformed using the depiction system. The performance data preferablycomprises at least the position data for the minimally invasiveprocedure, and preferably at least one of the 3D data, the orientationdata, the depiction data for the minimally invasive procedure or anycombination thereof. Preferably the performance data set comprises allthe data acquired during a minimally invasive surgery procedure. In anembodiment the performance data set further comprises calculated dataincluding the depiction date.

In an embodiment the performance data comprises at least one of theposition data, the 3D data, the orientation data or the depiction datafor the minimally invasive procedure.

In an embodiment the computer system is programmed to analyse therespective performance data set and to transmit a feedback evaluation(s)of the respective minimally invasive procedures to a display means.

In an embodiment the computer system is programmed to analyse therespective performance data set and preferably the computer system isconfigured to transmit a feedback evaluation(s) of the respectiveminimally invasive procedures to a display means, such as the displayunit and/or a printer.

Thereby a user—i.e. a surgeon or a person under training for becoming asurgeon—can in a very simple way receive feedback and he can evaluatehis improvement. Advantageously the computer system is configured totransmit a feedback evaluation(s) upon a request. Each performance dataset preferably has a unique code such that the respective performancedata sets can be retrieved for displaying of the depiction of the realtime relative spatial position(s) of the surgical tool e.g. for thesurgeon or for training purpose of other surgeons.

In an embodiment the computer system is programmed to determine animproved performance data set relative to a selected performance set,and to transmit at least a part of the data, such as depiction data ofthe improved performance data set to a display means, such as thedisplay unit and/or a printer.

Preferably the computer is programmed to compare the improvedperformance data set relative to the selected performance set and theselected performance set to determine data differences and to transmitthe data differences to a display means, such as the display unit and/ora printer.

Thereby the user can compare his performance with a superior performancedata set e.g. a calculated performance data set and he can retrieveinformation about what can be improved, where he uses superfluousmovements etc.

In an embodiment the computer system is programmed to determinedifference in performance data, categorizing the data and store thecatagorized data for machine learning purposes.

In an embodiment the computer system is programmed to transmit selecteddata of the one or more performance data set to a display means, such asthe display unit and/or a printer, preferably upon request from a user.

Thereby the respective performance data sets can be retrieved fordisplaying of the depiction of the real time relative spatialposition(s) of the surgical tool e.g. for use in education.

In an embodiment the depiction system is configured for operating in atraining mode. The surgical tool is a training tool, which may or maynot resemble a real surgical tool.

The depiction system when operating in its training mode is configuredfor determine training depiction data representing a training depictionof the real time relative spatial position(s) of the training tool ontoat least a portion of the surface contour of the surface section of theminimally invasive surgery cavity, and for transmitting the trainingdepiction data to the display unit for real time correlated depiction ofmovements of the training tool.

When the depiction system is in its training mode the minimally invasivesurgery cavity surface section may for example be an artificial surfacesection and in this embodiment the training tool may advantageously bean artificial tool.

In an embodiment the minimally invasive surgery cavity surface sectionwhen the depiction system is in its training mode is an actual minimallyinvasive surgery cavity. In this embodiment the training tool is a realsurgical tool. In principle the surgeons is constantly in training forimproving their technique in order to perform a minimally invasivesurgery as optimal as possibly—e.g. using the least movements of thesurgical tool possibly and/or performing the minimally invasive surgeryas fast as possible.

In an embodiment the position data generation means being configured forobtaining real time spatial position data of at least a part of thetraining tool and for transmitting the obtained spatial position data tothe computer system, the computer system being programmed for

-   -   calculating the training depiction data representing the        training depiction of the real time relative spatial position(s)        of the training tool onto at least a portion of the surface        contour of the surface section of the minimally invasive surgery        cavity, and for    -   transmitting the training depiction data to the display unit for        real time correlated depiction of movements of the training        tool.

The computer system is advantageously configured for transmitting thetraining depiction data and associated performance depiction data ofstored performance data set for the same or a corresponding minimallyinvasive surgery cavity to the display unit for real time correlateddepiction of movements of the training tool. Preferably the transmissionof the training depiction data and associated performance depiction dataare timely associated.

Thereby a user during training can benchmark against an earlierperformed minimally invasive surgery procedures or calculated proceduresor his own earlier procedures.

In an embodiment the computer system is configured for transmitting thetraining depiction data and associated performance depiction data ofstored performance data set for the same or a corresponding minimallyinvasive surgery cavity for performing a benchmarking of the performancedepiction data relative to said training depiction data.

In an embodiment the performance depiction data set is transmitted tothe display unit in a speed of about real-time speed.

In an embodiment the performance depiction data set is transmitted tothe display unit in a speed which is less than real-time speed.

Thereby a person under training can perform the minimally invasivesurgery procedure at a slower pace for training purpose and he canstep-by-step increase the pace to reach the real time speed or evenfaster if desired for training purpose.

In an embodiment the computer system is configured for receivingsupervisor input e.g. from a user or a training database, to determinesupervisor depiction data based on the supervisor input and transmittingthe supervisor depiction data to the display unit, preferably thecomputer is configured for acquiring the supervisor input from atraining database. The supervisor input may be input representingselected positions and movements of the training tool.

In an embodiment the superviser is a robot in data communication withthe computer system.

Advantageously the computer system is configured for transmitting thetraining depiction data and the associated performance depiction dataand/or the supervisor depiction data to the display unit to be displayedseparately, such as in a side by side configuration. Thereby the personunder training, such as a surgeon aiming to improve his skill can betrained to position the training tool in a selected position or movingthe training tool in a select way as instructed by the supervisor input.The person under training can try to move the training tool such thatthe training depiction follows associated performance and/or thesupervisor depiction.

In an embodiment the supervisor is another surgeon observing andadvising a surgeon during a minimally invasive surgery procedure. In anembodiment there are for example two surgeons operating together, whereone is handling the surgical tool and the other one is acting assupervisor making suggestions for positioning and movements of thesurgical tool and visa verse. It should be understood that the displayunit may be in form of two or more sub unit.

All features of the inventions including ranges and preferred ranges canbe combined in various ways within the scope of the invention, unlessthere are specific reasons not to combine such features.

BRIEF DESCRIPTION OF EXAMPLES

Preferred embodiments of the invention will be further described withreference to the drawings.

FIG. 1 is a schematic view of an embodiment of a depiction system of theinvention.

FIG. 2 is a schematic view of another embodiment of a depiction systemof the invention.

FIG. 3 is an example of a 3D database system—classification scheme.

FIG. 4 is an example of a 3D database system—3D data set scheme.

FIG. 5 is a schematic side view of a surgical instrument comprising asurgical tool.

FIG. 6 is a schematic side view of another surgical instrumentcomprising a surgical tool.

FIG. 7 is a schematic transverse cross sectional view of a minimallyinvasive surgery cavity and a number of sensors.

FIG. 8 is an illustration of an example of a real time correlateddepiction of movements of a surgical tool, wherein the depictioncomprises a dynamic changing of color scales correlated with movement ofthe not shown surgical tool.

FIG. 9 is an illustration of an example of a real time correlateddepiction of movements of a surgical tool, wherein the depictioncomprises a dynamic changing of pattern correlated with movement of thenot shown surgical tool.

FIG. 10 is an illustration of an example of a real time correlateddepiction of movements of a surgical tool, wherein the depictioncomprises a dynamic changing of light dots correlated with movement ofthe not shown surgical tool.

FIG. 11 is an illustration of an example of a real time correlateddepiction of movements of a surgical tool, wherein the depictioncomprises a dynamic changing of color dots correlated with movement ofthe not shown surgical tool.

FIG. 12 is an illustration of an example of a real time correlateddepiction of movements of a surgical tool, wherein the depictioncomprises a dynamic changing of rings correlated with movement of thenot shown surgical tool.

FIG. 13 is an illustration of an example of a real time correlateddepiction of movements of a surgical tool, wherein the depictioncomprises a dynamic changing of a perimeter and a bulge shaped markingcorrelated with movement of the not shown surgical tool and where anadditional supervisor instruction is depicted.

FIG. 14 is an illustration of an example of a depiction system of theinvention wherein the depiction comprises sound and/or the displaycomprises a goggle based display.

FIG. 15 is an illustration of an example of a depiction system of theinvention wherein the surgical tool comprising a pattern emittingprojector emitted onto the surface section of the minimally invasivesurgery and the 3D surface data generation means and/or the positiondata generation means and/or the orientation data generation meanscomprises a data collection system in data connection with or integratedwith the computer system, and where the data collection system comprisesan optical recorder.

FIG. 16 is an illustration of an example of a depiction system of theinvention wherein the surgical tool as well as the cannula/trocar eachcomprises a sensor, the depiction system comprises a data collectionsystem for collecting data from the sensors, a data collection systemfor collecting data from a reader and is further configured forgenerating a graphical depiction besides a real image of the relevantsurface section of the minimally invasive surgery.

FIG. 17 is an illustration of an example of real time correlateddepictions of movements of a surgical tool at 3 consecutive points intime, wherein the surgical tool is positioned with different distance tothe surface section in longitudinal distal direction to the surgicaltool.

FIG. 18 is an illustration of an example of a depiction system of theinvention comprising a data collection system for collecting data from asensor mounted onto the surgical tool, an optical recorder as well as anacoustic sensor, such as an ultrasound sensor, a wherein the depictionsystem is configured for generating a depiction onto a real image.

FIG. 19 is an illustration of an example of a depiction system of theinvention comprising a data collection system for collecting data fromnot shown sensors. The computer system stores a plurality of performancedata sets and is configured to benchmark against a selected performancedata set and to evaluate a minimally invasive surgery procedure by auser. The computer system is in digital connection with an additionaldisplay unit such as a smart phone or a printer for transmitting theevaluation.

FIG. 20 is an illustration of an example of a depiction system of theinvention comprising a data collection system for collecting data from arecorder and other not shown sensors. The computer system is in dataconnection with a supervisor control unit for receiving input. Thedepiction is displayed onto a display unit together with a real imageand a supervisor input.

FIG. 21 is an illustration of an example of a depiction system of theinvention comprising a data collection system for collecting data from arecorder and other not shown sensors. The computer system is in dataconnection with a robot controller for transmitting 3D surface data,real time spatial position data and real time orientation data to therobot controller. The robot controller is configured for controlling arobot for handling the surgical tool for performing a minimally invasivesurgery procedure and the depiction system display the depiction, whichcomprises a real time correlated depiction of movements of the surgicaltool by the robot.

The figures are schematic and are not drawn to scale and may besimplified for clarity. Throughout, the same reference numerals are usedfor identical or corresponding parts.

The depiction system illustrated in FIG. 1 comprises a computer system1, 3D surface data generation means 2 and position data generation means3 in data connection with the computer system 1, e.g. as illustratedwith wires 5 or wireless for feeding 3D surface data and real timeposition data to the computer system 1. The computer system 1 is hereillustrated as one unit, but as explained the computer system couldcomprise two or more units in data communication depiction system of theinvention.

The computer system 1 is programmed for

-   -   determining a 3D surface contour of at least a part of the        target area of an not shown surface section of the minimally        invasive surgery cavity using the 3D data,    -   determining real time spatially position(s) of a not shown        surgical tool relative to at least a part of the target area of        the at least one surface section using the spatial position data        and the surface position data,    -   calculating depiction data representing a depiction of the real        time relative spatial position(s) of the surgical tool onto at        least a portion of the surface contour of the surface section of        the minimally invasive surgery cavity, and for    -   transmitting the depiction data to the display unit 4 for real        time correlated depiction of movements of the surgical tool.

The display unit can be as described above.

The depiction system illustrated in FIG. 2 also comprises a computersystem 11 e.g. as the computer system of FIG. 1. The depiction system ofFIG. 2 differs from the depiction system of FIG. 1 in that it comprisesa combined 3D surface data generation means and position data generationmeans 12 and in that it comprises a display unit in form of two displaysub unit 14 a, 14 b. Advantageously the real time correlated depictionof movements of the surgical tool is displayed on one of the sub unit 14a and real images e.g. live images and/or benchmarking depiction isdisplayed on the other sub unit 14 b.

FIG. 3 illustrates a classification scheme which may be comprised in a3D database system. As indicated each classification has a unique code,linking each classification to one or more 3D data sets. Eachclassification set is classified in accordance with a classification setcomprising a number of patient characteristics as indicated in thescheme of FIG. 3 and comprises for example age, gender, weight, height,body circumferential dimension(s) or any combinations thereof. Theselected characteristics depend largely on the type of minimallyinvasive surgery cavity in question.

FIG. 4 illustrates a scheme of 3D data sets which may be comprised in a3D database system, wherein each 3D data set is associated to to apatient characteristic as indicated with the unique codes, correspondingto the unique codes of FIG. 3.

The surgical instrument shown in FIG. 5 is a laparoscopic instrument. Itshould be understood that in the invention other surgical instrumentwith surgical tool could as well be applied.

The surgical instrument has a handle portion 22 and a body tool portion23 with a surgical tool portion 24 in the present case forceps. Thesurgical tool portion 24 and the part of the body portion 23 adapted tobe inserted into the minimally invasive surgery cavity is referred to asthe surgical tool. In other words the surgical instrument 21 comprisesthe surgical tool 23, 24 and the handle 22. The body portion 23interconnect the handle portion 22 which is arranged at the proximal endof the surgical instrument and the surgical tool portion 24, which isarranged at the distal end of the surgical instrument. The body portion23 is arranged in the distal/proximal direction, which is also referredto as the longitudinal direction of the surgical tool 23, 24.

In another embodiment the surgical tool portion 24 may be anothersurgical tool portion e.g. a grasper, a suture grasper, a stapler, adissector, scissors, a suction instrument, a clamp instrument, anelectrode, a curette, ablators, scalpels, a needle holder, a biopsy andretractor instrument or a combination thereof.

The surgeon operate the surgical instrument 21 by holding the handleportion 22 and can in this way control the surgical tool and by pressingor manipulating the handle portion the forceps can be controlled.

The surgical instrument 21 further comprises a first sensor 28 a, such asensor as described above, mounted to the surgical tool and a secondsensor 28 b mounted at the handle. The sensor may be linked to the notshown data collecting system of the computer system e.g. by wire (forexample optical fiber(s) or wireless e.g. blue tooth. The two sensorsmay provide both real time position data as well as real timeorientation data.

In an alternative embodiment the surgical instrument may comprise apattern generating arrangement. The surgical tool with a handle may forexample be in form of a surgical instrument assembly as described inWO15124159. The reflected/scattered pattern may thus be collected by areader and transmitted to a data collecting system of the computersystem to provide a part of the 3D surface data, which may preferablyalso be real time 3D data and the real time position data and preferablyalso real time orientation data.

FIG. 6 illustrates a variation of the surgical instrument of FIG. 5,where the pair of sensors 28 c is positioned on the graspers of theforceps of the surgical tool portion 24 of the surgical tool. Therebythe movements of the graspers may be monitored as well and at the sametime the sensors 28 c on the graspers may be used to determine the realtime position data and the real time orientation data.

FIG. 7 illustrates a minimally invasive surgery cavity 31—here theabdominal cavity of a patient 36. The minimally invasive surgery cavity31 is seen in a transverse cross sectional view. The anatomic plans of ahuman patient are indicated in the image 30 and the minimally invasivesurgery cavity 31 is seen in the transverso plan from the top verticalview.

The patient 36 is positioned on an operation table 33 with his back 34against the table 33 and his front 35 upwards. The minimally invasivesurgery cavity is blown up by injecting a gas into the cavity through anot shown incision and the surface is indicated with reference 32. Asillustrated the minimally invasive surgery cavity surface 32 may be veryuneven and with large curved projections and recesses.

A number of sensors 38 a are positioned onto the operation table. Only 4sensors 38 a are shown, but preferably at least 4 sensors 38 a arepositioned in a rectangular configuration on the operation to define anX-Y-Z plan and optionally with additional sensors.

Further a number of sensors 38 b are mounted to the patient e.g. withinthe minimally invasive surgery cavity 31 as shown.

The sensors 38 a, 38 b may e.g. be as described above.

An example of a real time correlated depiction of movements of asurgical tool is illustrated in FIG. 8. The depiction comprises a numberof squared sections 41, 42, 43 concentrically arranged to surround acentral area 40. The dynamic depiction may advantageously change in acorrelation with movement of a not shown surgical tool, e.g. by changingthe width of the respective squared sections 41, 42, 43 individuallyfrom each other, by changing the corner angles of the respective squaredsections 41, 42, 43 individually from each other, by changing the colorand/or color pattern of the respective squared sections 41, 42, 43individually from each other, by changing the size and/or shape of thecentral area 40 or by any combinations thereof. For example the size ofthe central area 40 may indicate the distance between the surgical tooland the minimally invasive surgery cavity surface section e.g. seen inlongitudinal distal direction from the surgical tool. The color or colorvariations along the square shape of one or more of the respectivesquared sections 41, 42, 43 may indicate the orientation of the tool,and the width and/or the corner angles of the respective squaredsections 41, 42, 43 may indicate the contour of the cavity surfacesection. When the surgical tool is moved the depiction will be changingin a way which is correlated with movement of the surgical tool.

Another example of a real time correlated depiction of movements of asurgical too is illustrated in FIG. 9. The depiction is displayed on adisplayer 50 in form of a flat screen. The depiction comprises a dynamicchanging of pattern correlated with movement of the not shown surgicaltool. The pattern is a crosshatched pattern 51 comprising a grid oflines which when the surgical tool is far from the surface section hasparallel, straight and crossed lines, whereas when the surgical tool iswithin a selected distance from the surface section the lines arebending as indicated with the reference 53 in dependence on the contourof the surface. At the same time the distance between parallel lines mayreflect the distance between the surgical tool and the minimallyinvasive surgery cavity surface section e.g. seen in longitudinal distaldirection from the surgical tool. Further the angles between thecrossing lines may indicate the orientation and angulation of thesurgical tool. When the surgical tool is moved the depiction will bechanging in way which is correlated with movement of the surgical tool.

A further example of a real time correlated depiction of movements of asurgical tool is illustrated in FIG. 10. The depiction is displayed on adisplayer 60 in form of a flat screen. The depiction comprises a dynamicchanging of light dots 61 correlated with movement of the not shownsurgical tool. The light dots 61 are illustrated in their home positionwhere the surgical tool is far from the surface section and the lightpattern is very regular. When the surgical tool is within a selecteddistance from the surface section the position of the light dots arechanging in dependence on the contour of the surface. At the same timethe size of the light dots may reflect the distance between the surgicaltool and the minimally invasive surgery cavity surface section e.g. seenin longitudinal distal direction from the surgical tool. Further thenumber of light dots or the relative sizes of light dots may indicatethe orientation and angulation of the surgical tool. When the surgicaltool is moved the depiction will be changing in way which is correlatedwith movement of the surgical tool.

A variation of the depiction of FIG. 10 is illustrated in FIG. 11. Herethe light dots 71 are arranged in another home configuration and thedots 71 may further have varying colours e.g. in dependence of thedistance to the target area.

The time correlated depiction of movements of a surgical toolillustrated in FIG. 12 comprises a dynamic changing of rings 81, 82, 83,84 correlated with movement of the not shown surgical tool. The rings81, 82, 83, 84 are concentrically arranged and may vary in size, inshape, in line thickness, in line color, in individual distances and orin other in way to indicate surface contour, distances, orientation andother information which may be relevant for the user e.g. as describedabove.

The time correlated depiction of movements of a surgical toolillustrated in FIG. 13 comprises a dynamic changing of a perimeter 91and a bulge shaped marking 92 correlated with movement of the not shownsurgical tool. The shape, size, line thickness and/or color of theperimeter 91 may e.g. change in dependence of the contour of the surfacesection the distance between the surgical tool and the surface sectionand/or the orientation of the surgical tool and the bulge shaped marking92 may e.g. change in dependence of the distance between the surgicaltool and the target area and optionally of the orientation of thesurgical tool relative to the target area. The dot 93 indicates a markerfor the longitudinal direction of the not shown surgical tool.

FIG. 14 illustrates at least a part of a depiction system comprising acomputer system where a computer 101 and a data collection system 102 ofthe depiction system are shown. The data collection system 102 isconfigured for collecting the various data comprising at least 3Dsurface data and real time position data. The collected data istransmitted to the computer 101 for calculating depiction data and thedepiction data is transmitted to one or more display units comprising ascreen 103, a loud speaker 104 for displaying a sound depiction e.g. abeep—beep sound as explained above and/or goggles 105 for displaying ona wearable display unit.

As it can be understood the display and the depiction may take manydifferent forms.

The depiction system of FIG. 15 is a variation of the depiction systemof FIG. 14 where only the screen display 103 is indicated. The depictionsystem comprises an endoscope 108 and a surgical instrument with a notshown handle e.g. forming part of or integrated with a robot arm and asurgical tool 107 configured for emitting a light pattern.

The surgical tool may have a handle for example be in form of a surgicalinstrument assembly as described in WO15124159.

The surgical tool is inserted through a not shown incision through theskin layer 106 of a patient and into the minimally invasive surgerycavity 100 a. The light pattern is emitted towards the relevant surfacesection 100 b and the light pattern 109 is impinging on and a part isreflected and/or scattered from the surface section 100 b of theminimally invasive surgery cavity 100 a. The endoscope comprises anoptical recorder for recording the reflected and/or scattered lightpattern 109, and the collected data which advantageously includes 3Dsurface data (in real time), real time position data and real timeorientation date is transmitted by wire or wireless to the datacollection system 102. In a variation thereof the optical recorder 108is not a part of an endoscope but inserted through the skin layer 106 atanother place than the site of the endoscope entry. In another variationthe optical recorder 108 is fixed to or integrated with the surgicaltool.

The depiction system illustrated in FIG. 16 comprises a computer systemwhere a computer 111 and a data collection system of the depictionsystem are shown. The data collection systems 112 a is configured forcollecting the various data comprising at least 3D surface data and realtime position data and to transmit the data to the computer 111 forcalculating depiction data and the depiction data is transmitted to oneor more display units comprising a screen 113.

In the illustrated embodiment the depiction system comprises anendoscope 118 and a surgical instrument with a handle 117 a and asurgical tool 117. The surgical tool may further emit a light pattern asdescribed in FIG. 15.

The endoscope 118 comprises a recorder for recording real images of thesurface section and the surgical tool. The real images are collected ina secondary data collection system 112 b of the computer system andtransmitted to the computer from where they are transmitted fordisplaying on the screen 113 such that the real images 115 are timelyassociated with the depiction 119.

In the shown FIG. 16 the surgical instrument is inserted through anincision in the skin layer 116 of a patient via a cannula/trocar 117 bwhich generates an access port for the surgical tool 117. A sensor S2 ismounted to the surgical tool 117 and another sensor S1 is mounted to thecannula 117 b. A target area T is marked at the surface section 110 forillustrative purposes.

The computer has information about the position of this target area Te.g. via the images recorded by the recorder 118 of by other means. Therespective sensors S1 and S2 are advantageously distance sensorsconfigured for determining the respective distances D1 and D2 to thetarget area T and preferably also the respective position of the sensorsS1, S2 relative to the target area to thereby determine the orientationof the surgical tool 117. Optionally, the longitudinal direction of thesurgical tool 117 is marked/depicted by a dot at target area T.

The real images 115 and the depiction are displayed on the display 113besides each other. The target area T may also be shown on the realimage 115. The depiction 119 is a graphical depiction showing thedistance D1 between sensor S1 and the target area T as well as thedistance D2 between sensor S2 and the target area T. e.g. by a barindicator 199 a that is moved in horizontal direction.

FIG. 17 illustrates a real time correlated depictions of movements of asurgical tool 127 at 3 consecutive points in time T1, T2, T3, whereinthe surgical tool is positioned with different distance D1, D2, D3 tothe surface section 120 e.g. a target area of the surface section inlongitudinal distal direction to the surgical tool. As it can be seenthe depiction 129 on the real image 125 at the time T1 where thedistance D1 is relatively large, is accordingly relatively large. At thetime T2 the distance D2 is smaller than D1 and the depiction 129 isaccordingly smaller. At the time T3 the distance D3 is smaller than D2and the depiction 129 is accordingly even smaller. The displayed size ofthe real image 125 is kept substantially constant.

The depiction system illustrated in FIG. 18 comprises a computer systemwhere a computer 131 and a data collection system 132 of the depictionsystem are shown. The data collection systems 132 is configured forcollecting the various data comprising at least 3D surface data and realtime position data and to transmit the data to the computer 131 forcalculating depiction data and the depiction data is transmitted to oneor more display units comprising a screen 133. The depiction systemfurther comprises an endoscope 138, an acoustic sensor 134, such as anultrasound sensor, and a surgical instrument with a handle 137 a and asurgical tool 137. The surgical tool may further emit a light pattern asdescribed in FIG. 15.

The endoscope 138 comprises an arrangement for emitting a light patternand a recorder for recording real images of the surface section and thesurgical tool 137. The real images are collected in the data collectionsystem 132 of the computer system and transmitted to the computer 131from where they are transmitted for displaying on the screen 133 suchthat the real images 135 are timely associated with the depiction 139.

In the shown FIG. 18, the surgical instrument is inserted through anincision in the skin layer 136 of a patient via a cannula 137 b whichgenerates an access port for the surgical tool 137. A sensor S1 ismounted to the surgical tool 137 for collection real time position dataand transmitting the data to the data collection system. The endoscope138 is inserted through another incision in the skin layer 136 of thepatient. The endoscope is emitting a stationary pattern such as acrosshatched pattern, which is impinging onto and at least partlyreflected/scattered from the surface section 130 of the cavity, therebyrevealing the surface contour of the surface section which is recordedboth in form of 3D surface data and in form of real images by therecorder of the endoscope 138. The acoustic sensor 134 is insertedthrough a further incision through the skin layer 136 of the patient forrecording additional 3D surface data. All the recorded data and imagesare transmitted to the data collecting system 132. The data and imagesare transmitted to the computer 131 for calculating depiction data andthe depiction data and the images are in a timely associate fashiontransmitted to the display 133 for being displayed, where the realimages 135 are displayed and a part of the depiction 139 a are displayedon top of the real images 135. The part of the depiction 139 a displayedon top of the real images 135 are advantageously at least partlytransparent for the real images 135. The depiction also comprises agraphical depiction part 139 b e.g. in form of a distance indicationdisplayed beside the real images for example indicating the distancebetween the surgical tool and a target area.

The depiction system illustrated in FIG. 19 comprises a data collectionsystem 142 and a computer 141 of a depiction system. The data collectionsystems 142 is configured for collecting the various data from not shown3D surface data generation means, position data generation means andoptionally other means as described above, where the data comprises atleast 3D surface data and real time position data. The data collectionsystems 142 is further configured for collecting real images asdescribed above and for transmitting the data and images to the computer141 for calculating depiction data and the depiction data and the imagesare in a timely associate fashion transmitted to the display 143 a,where at least a part of the depiction 149 a are displayed on top of thereal images 145. The part of the depiction 149 a displayed on top of thereal images 145 are advantageously at least partly transparent for thereal images 145. The computer stores a number of performance data setsas explained above and is programmed to analyse and optionally benchmarkthe performance of a user relatively to one or more stored performancedata set—e.g. generated by the same user for determine his improvement.In the example in FIG. 19 the computer system has transmitted the userscore 149 b for being displayed. The depiction system is further digitalconnected to a printer, other display unit and/or a smart phone 143 bfor printing or displaying a full evaluation of a user performance of aminimally invasive surgery procedure. The full evaluation may includeboth timely benchmarking and spatially surgical tool movementbenchmarking as well as any other benchmarking.

The depiction system illustrated in FIG. 20 comprises a data collectionsystem 152 and a computer 151 of a depiction system. The data collectionsystems 152 is configured for collecting the various data from various3D surface data generation means, position data generation means andoptionally other means e.g. from an endoscope recorder 158 and/or sensoron a surgical tool 157 as described above, where the data comprises atleast 3D surface data and real time position data. The data collectionsystems 152 is further configured for collecting real images asdescribed above and for transmitting the data and images to the computer151 for calculating depiction data and the depiction data and the imagesare in a timely associate fashion transmitted to the display 153, whereat least a part of the depiction 159 are displayed on top of the realimages 155. The part of the depiction 159 displayed on top of the realimages 155 are advantageously at least partly transparent for the realimages 155.

The depiction system further comprises a supervisor control unit 154 ain communication with or comprised in its computer system. Thesupervisor control unit comprises a digital user (supervisor) interfaceand/or a sound recorder for recording supervisor input, such assupervisor instructions. The supervisor input data are transmitted viathe computer 151 to the display 153 for being displayed as a supervisordepiction 154 b.

The depiction system illustrated in FIG. 21 a data collection system 162and a computer 161 of a depiction system. The data collection systems162 is configured for collecting the various data from various 3Dsurface data generation means, position data generation means andoptionally other means e.g. as explained herein e.g. below. The datacomprises at least 3D surface data and real time position data. The datacollection systems 162 may further be configured for collecting realimages as described above and for transmitting the data and images tothe computer 161 for calculating depiction data and the depiction dataand the images are in a timely associate fashion transmitted to thedisplay 163.

In the illustrated embodiment the data generation means comprises anendoscope 168 comprising an arrangement for emitting a light pattern anda recorder for recording reflected/scattered light from the pattern andoptionally for recording real images of the surface section and thesurgical tool 167. In a variation of the shown embodiment thearrangement for emitting a light pattern and a recorder for recordingreflected/scattered light from the pattern is not a part of an endoscope168 but inserted through the skin layer 166 at another place than thesite of the endoscope entry. The surgical tool 167 of the depictionsystem is also configured for emitting a light pattern.

The surgical tool with a handle may for example be in form of a surgicalinstrument assembly as described in WO15124159. The handle and/or thewhole surgical tool may for example be a part of a robotic arm.

The endoscope 168 is inserted through an incision in the skin layer 166of the patient. The endoscope is emitting a stationary pattern such as acrosshatched pattern 168 a, which is impinging onto and at least partlyreflected/scattered from the surface section of the cavity, therebyrevealing the surface contour of the surface section which is recordedboth in form of 3D surface data and preferably also in form of realimages by the recorder of the endoscope 168.

The surgical tool 167 is inserted through another incision (or sameincision) through the skin layer 166 of the patient and into theminimally invasive surgery cavity. The light pattern is emitted towardsthe relevant surface section and the light pattern 167 a is impinging onand at least partly reflected and/or scattered from the surface sectionof the minimally invasive surgery cavity. As the surgical tool 167 ismoved the pattern 167 a emitted from the surgical tool 167 becomes adynamic pattern 167 a upon the stationary pattern 168 a from theendoscope 168. As the reflected and/or scattered light is recorded bythe recorder of the endoscope 168 large amounts of 3D surface data, realtime position data as well as real time orientation data is obtained andtransmitted to the data collection system 162. The movements of thedynamic pattern may be for example be related to the stationary patternand the relevant 3D, position and orientation data may be determinedusing trigonometric calculation methods.

The depiction system further comprises a robot controller 160. Thecomputer 161 is in data connection with the robot controller 160 fortransmitting at lease a part of the collected data including 3D surfacedata, real time spatial position data and real time orientation data tothe robot controller 160. The robot controller 160 is configured forcontrolling a not shown robot for handling the surgical tool forperforming a minimally invasive surgery procedure and the depictionsystem is configured for displaying the depiction, which comprises areal time correlated depiction of movements of the surgical tool by therobot. Thereby a supervisor e.g. an operator can keep the robot underobservation during its performing of the minimally invasive surgery viathe displayed depiction and thereby control that the robot is operationsufficiently accurate or as explained, the supervisor may correct therobot by feeding instructions to the robot controller e.g. via asupervisor control unit as shown in FIG. 20.

The figures are schematic and are not drawn to scale and may besimplified for clarity. Throughout, the same reference numerals are usedfor identical or corresponding parts.

What is claimed is:
 1. A depiction system for generating a real timecorrelated depiction of movements of a surgical tool, the systemcomprising a computer system, 3D surface data generation means adaptedfor providing the computer system with three-dimensional (3D) datarepresenting at least one surface section in 3D space of a minimallyinvasive surgery cavity, wherein said surface section comprises a targetarea, and position data generation means adapted for obtaining real timespatial position data of at least a part of the surgical tool and fortransmitting said obtained spatial position data to said computersystem, wherein at least one of said 3D surface data generation meansand said position data generation means comprises surface position datageneration means adapted for providing surface position data of at leastsaid target area, said computer system being programmed for determininga 3D surface contour of at least a part of said target area of saidsurface section of said minimally invasive surgery cavity using said 3Ddata, determining real time spatially position(s) of said surgical toolrelative to at least a part of said target area of said at least onesurface section using said spatial position data and said surfaceposition data, calculating depiction data representing a depiction ofsaid real time relative spatial position(s) of said surgical tool ontoat least a portion of said surface contour of said surface section ofsaid minimally invasive surgery cavity, and for transmitting saiddepiction data to a display unit for real time correlated depiction ofmovements of said surgical tool.
 2. The depiction system of claim 1,wherein the 3D surface data generation means comprises a 3D databasesystem, the 3D database system comprising at least one 3D data set forat least one surface section of each of a plurality of classifiedminimally invasive surgery cavities, wherein each 3D data set isassociated to said respective surface section(s) of at least one of saidclassified minimally invasive surgery cavities, and wherein saidcomputer is configured for acquiring at least one 3D data set for atleast one surface section of a classified minimally invasive surgerycavity.
 3. The depiction system of claim 2, wherein one or more of said3D data sets comprises estimated 3D data, calculated 3D data, measured3D data or any combination thereof, said 3D data sets each is associatedto a patient characteristic.
 4. The depiction system of claim 1, whereinthe 3D surface data generation means comprises a 3D surface sensorsystem for determining at least a part of said 3D data for at least saidsurface section of said minimally invasive surgery cavity andtransmitting means for transmitting said determined for 3D data to saidcomputer.
 5. The depiction system of claim 4, wherein the sensor systemfor determining at least a part of said 3D data for at least saidsurface section of said minimally invasive surgery cavity is configuredfor generating at least one of pre-operative data or intra-operativedata.
 6. The depiction system of claim 4, wherein the 3D surface sensorsystem comprises at least one local reference sensor.
 7. The depictionsystem of claim 4, wherein the 3D surface sensor system comprises a 3Doptical sensor system comprising at least one optical source and atleast one optical reader and wherein the at least one optical source isconfigured for emitting an optical tracking light pattern which patternwhen impinged onto and reflected and/or scattered from the surfacereveal the contour of the surface to thereby provide 3D data to berecorded by the optical reader.
 8. The depiction system of claim 7,wherein said at least one of said optical pattern emitting source andsaid recorder being positioned on an endoscope.
 9. The depiction systemof claim 7, wherein the optical source is adapted for emitting a dynamicoptical pattern onto at least a part of the surface section.
 10. Thedepiction system of claim 7, wherein the depiction system comprises botha stationary and a dynamic optical pattern source, said computer systembeing configured for generating at least a part of said 3D data and saidreal time position data simultaneously.
 11. The depiction system ofclaim 1, wherein the position data generation means for obtaining realtime spatial position data of at least a part of the surgical toolcomprises a position sensor system, said position sensor systemcomprises a 3D optical sensor, an acoustic sensor, a magnetic sensor, anelectric sensor, an accelerometer, a gyroscope, a gravimeter, aninertial navigation system, a local positioning system or anycombinations thereof.
 12. The depiction system of claim 11, wherein saiddepiction system comprises a robot controller in data communication withsaid position sensor for receiving position data and for using saidposition data for controlling a robot for handling said surgical tool.13. The depiction system of claim 11, wherein said position sensorsystem is configured for obtaining real time spatial position data ofthe surgical tool and for transmitting said obtained spatial positiondata to said computer system in the form of real time spatial positiondata in an X-Y-Z.
 14. The depiction system of claim 11, wherein saidreal time spatial position data comprises position data correlated tosaid 3D data representing at least one surface section in 3D space andsaid real time spatial position data comprises a distance from thesurgical tool of said at least one part of said surgical tool to saidtarget area of said surface section.
 15. The depiction system of claim14, wherein said real time spatial position data comprises position datacomprises a distance from said surgical tool to a critical structure.16. The depiction system of claim 1, wherein said system furthercomprises orientation data generation means for obtaining real timespatial orientation data of at least a part of the surgical tool and fortransmitting said obtained spatial orientation data to said computersystem, said computer system being programmed for determining real timespatially orientation(s) of said surgical tool using said spatialorientation data, calculating depiction data representing a depiction ofsaid real time spatial orientation(s) of said surgical tool onto atleast a portion of said surface contour of said surface section of saidminimally invasive surgery cavity, and for transmitting said depictiondata to said display unit for real time correlated depiction ofmovements of said surgical tool.
 17. The depiction system of claim 16,wherein said depiction data representing a depiction of said real timerelative spatial position(s) and said real time spatial orientation(s)of said surgical tool onto at least a portion of said surface contour ofsaid surface section of said minimally invasive surgery cavity comprisesdepiction data representing an associated depiction of said real timerelative spatial position(s) and said real time spatial orientation(s)of said surgical tool onto at least a portion of said surface contour.18. The depiction system of claim 15, wherein said orientation datageneration means comprises an orientation sensor.
 19. The depictionsystem of claim 1, wherein said depiction data representing a depictionof said real time relative spatial position(s) of said surgical toolonto said determined 3D surface contour of said surface section of saidminimally invasive surgery cavity comprises depiction data encoding adepiction of a dynamic pattern representation, a dynamic scaling ofcolours representation, a dynamic schematic representation, a dynamicgraphical representation and/or a dynamic augmented realityrepresentation.
 20. The depiction system of claim 19, wherein saidencoded depiction comprises a non-image-accurate depiction of thesurgical tool.
 21. The depiction system of claim 19, wherein saiddepiction data comprises data encoding a depiction of a dynamic patternrepresentation, wherein the dynamic pattern representation comprises adepiction of a virtual pattern resembling an emitted light patternimpinged onto the determined 3D surface contour, wherein the virtualpattern preferably comprises arch shaped and/or ring shaped lines and/ora plurality of angled lines.
 22. The depiction system of claim 19,wherein the dynamic pattern representation comprises a depiction of saidvirtual pattern onto the determined 3D surface contour, to provide thatthe depiction comprises a dynamic modification of the virtual patternwherein the dynamic modification is correlated to the determined realtime spatially position(s) of said surgical tool.
 23. The depictionsystem of claim 19, wherein said depiction comprises a depiction of adynamic augmented reality representation, wherein the dynamic augmentedreality representation comprises an augmented reality representation ofthe determined real time spatially position(s) of said surgical toolrelative to said determined 3D surface contour wherein the augmentedreality representation is dynamically modified in correlation to changesof the spatial position and orientation caused by the movements of thesurgical tool relative to said determined 3D surface contour.
 24. Thedepiction system of claim 15, wherein said depiction comprises a sounddepiction.
 25. The depiction system of claim 15, wherein said depictioncomprises a vibration depiction, comprising a vibration of the surgicaltool.
 26. The depiction system of claim 1, wherein said depiction systemcomprises a real imaging system configured for generating real imagingdata for a real imaging of said at least one surface section of saidminimally invasive surgery cavity, said real imaging system comprises a2D real imaging system, a 3D real imaging system, a virtual reality realimaging system or an augmented reality real imaging system.
 27. Thedepiction system of claim 21, wherein the depiction system is configuredfor displaying said real imaging of the at least one surface section ofthe minimally invasive surgery cavity and onto said real imaging todisplay said depiction of said real time relative spatial position(s) ofsaid surgical tool onto said surface contour of said surface section ofsaid minimally invasive surgery cavity.
 28. The depiction system ofclaim 1, wherein said depiction system comprises a robot controller indata connection with or integrated with said computer system, said robotcontroller being configured for receiving said respective data and forcontrolling a robot for handling said surgical tool for performing aminimally invasive surgery procedure.
 29. The depiction system of claim28, wherein depiction system being configured for, controlling saidrobot for handling said surgical tool for performing a minimallyinvasive surgery procedure and for transmitting said depiction data tosaid display unit, wherein said depiction comprises a real timecorrelated depiction of said movements of said surgical tool by saidrobot.
 30. The depiction system of claim 1, wherein said computer systemcomprises a memory and is configured for storing performance data sets,each performance data set comprises performance data associated with aminimally invasive procedure.
 31. The depiction system of claim 30,wherein said performance data comprises at least one of said positiondata, said 3D data, said orientation data or said depiction data forsaid minimally invasive procedure.
 32. The depiction system of claim 30,wherein said computer system is programmed to analyse said respectiveperformance data set and to transmit a feedback evaluation(s) of saidrespective minimally invasive procedures to a display means.
 33. Thedepiction system of claim 30, wherein said computer system is programmedto determine difference in performance data, categorizing the data andstore it for machine learning purposes.
 34. The depiction system ofclaim 1, wherein said computer system is configured for receivingsupervisor input, to determine supervisor depiction data based on saidsupervisor input and transmitting said supervisor depiction data to saiddisplay unit.
 35. The depiction system of claim 34, wherein saidcomputer is configured for acquiring said supervisor input from atraining database, via a digital user interface or by oral supervisorinput.
 36. The depiction system of claim 34, wherein said computer isconfigured for acquiring said supervisor input in the form of supervisorinput representing selected positions and movements of the trainingtool.
 37. The depiction system of claim 1, wherein said computer isprogrammed to receive instructions comprising classification data forselecting several 3D data sets and based on this instruction toacquiring selected 3D data sets and process the acquired 3D data sets todetermine the resulting 3D data which is applied to determine the 3Dsurface contour of the surface section of the minimally invasive surgerycavity.
 38. The depiction system of claim 1, wherein the determining ofsaid real time spatial position data comprises determining a distancefrom the surgical tool to a critical structure of the surface section.